DOCUMENT RESUME 



ED 417 269 



CE 075 658 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Radtke, Paul H. ; Frey, Paul R. 

Sea Stories: A Collaborative Tool for Articulating Tactical 
Knowledge . 

1997-12-00 

12p . ; Paper presented at the Interservice/Industry Training, 
Simulation, and Education Conference (Orlando, FL, December 
1-4, 1997) . 

Reports - Descriptive (141) -- Speeches/Meeting Papers (150) 

MF01/PC01 Plus Postage. 

Adult Education; Computer Oriented Programs; Computer Uses 
in Education; *Decision Making; * Instructional Development; 
*Knowledge Representation; *Military Training; *Multimedia 
Instruction; *Needs Assessment 
Scenarios; Story Boards; ^Subject Specialists 



ABSTRACT 



Having subject matter experts (SMEs) identify the skills and 
knowledge to be taught is among the more difficult and time-consuming steps 
in the training development process. A procedure has been developed for 
identifying specific tactical decision-making knowledge requirements and 
translating SME knowledge into appropriate multimedia representations. The 
procedure, which is called Sea Stories, is based on construction and analysis 
of a scenario by one or more SMEs. The Sea Stories procedure allows a team of 
domain experts to "articulate" their knowledge by describing a scenario 
(their sea story) in a series of computer-based storyboards. The major 
knowledge elicitation tasks performed by the SMEs are as follows: construct a 
scenario illustrating a particular problem or procedure; test the scenario's 
logic by identifying key variables in the scenario and their relationships; 
identify the knowledge required of decision makers to perform successfully in 
the scenario; and provide graphical, auditory, and verbal representations of 
the key knowledge elements. Possible storyboards include spatial situation 
overviews, team interaction diagrams, task flowcharts, and equipment 
diagrams. The Sea Stories procedure will be automated by using a suite of 
tools that are largely available as commercial off-the-shelf products. (MN) 
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SEA STORIES: A COLLABORATIVE TOOL FOR ARTICULATING 

TACTICAL KNOWLEDGE 

Paul H. Radtke 

Naval Air Warfare Center Training Systems Division 
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Among the more difficult and time-consuming steps in the training development 
process is the elicitation from subject matter experts (SMEs) of the skills and 
knowledge to be taught. As the use of advanced multimedia training technology has 
become more common, training development increasingly involves translating SME 
knowledge into appropriate media representations. This paper describes a procedure 
for identifying specific tactical decision making (TDM) knowledge requirements, and 
possible media-based representations of that knowledge. The intent of this procedure 
is to provide the basis for constructing tactical training documents using multimedia 
technology. The procedure, called ories, is built around the construction and 
analysis of a scenario by one or more SMEs. Sea Stories allows a team of domain 
experts to “articulate” their knowledge by describing a scenario (their sea story) in a 
series of computer-based storyboards. These storyboards include, for example, 
spatial situation overviews, team interaction diagrams, task flow charts, and equipment 
diagrams; and are integrated though a detailed timeline. Applied training research 
provides knowledge frameworks that can be used to guide and prompt experts to 
identify and refine components of the knowledge. The storyboards provide the basis 
for identifying these knowledge requirements, and the media representations that are 
associated with a tactical problem. Computer Supported Collaborative Work (CSCW) 
technologies facilitate communication among groups of subject matter experts using 
annotation techniques and revision control and tracking. 
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BACKGROUND AND OBJECTIVES 

Among the more difficult and time- 
consuming steps in the training development 
process is the elicitation from subject matter e x- 
perts (SMEs) of the skills and knowledge to be 
taught. Despite the variety of techniques for eli c- 
iting knowledge from SMEs that have been de- 
veloped, knowledge elicitation (KE) remains one 
of the primary “bottlenecks” in the process. 

All KE methods involve tradeoffs ( Cooke, 
1994). These tradeoffs typically balance the 
depth against the breadth of the knowledge eli c- 
ited from the SME. Consequently, it is necessary 
to tailor the method to the content domain and to 
the ultimate use to which the products of the 
analysis will be put. Declarative knowledge is 
typically extracted as verbal information through 
structured or unstructured interviews, think-aloud, 
or retrospective protocols. Procedural knowledge 
is usually extracted by analyzing actual task per - 
formance of experts on real or synthetic tasks. 

As the use of advanced multimedia training 
technology has become more common, training 
development increasingly involves translating 
SME knowledge into appropriate media repr e- 
sentations. The translation is complicated by the 
fact that most KE methods produce verbal repr e- 
sentations of knowledge that are often several 
conceptual steps removed from the form and 
context of the SME’s original articulation. When 
the content consists of context-independent d e- 
clarative knowledge, this translation may not si g- 
nificantly alter the validity of the analysis pro d- 
ucts. However, when the knowledge to be elicited 
is procedural in nature, is embedded in the co n- 
text in which it is to be used, or when the know I- 
edge is typically represented in non-verbal, per - 
ceptual forms, the translation of the SME’s output 
may significantly affect the completeness and 
accuracy of the analysis and the resulting media 
representations. This problem suggests the need 



to explore alternative analysis methods that pr e- 
serve the context and non-verbal aspects of the 
elicited knowledge. 

This discussion describes a procedure for 
identifying specific tactical decision making 
(TDM) knowledge requirements, and possible 
media-based representations of that knowledge. 
The intent of this procedure is to provide the basis 
for constructing electronic tactical training doc u- 
ments using multimedia technology. The intent of 
the KE procedure is to 

• Build on procedures and practices already 
used by SMEs; 

• Keep the amount of time or effort needed to 
perform the procedure in approximate proportion 
to the scope of the final product; 

• Ensure that the products of the procedure 
conform to standards of validity, consistency, ac- 
curacy, and completeness; and 

• Provide the basis for identifying and co n- 
structing effective multimedia objects and pre s- 
entation strategies. 

The procedure, called Sea Stories, is built 
around the construction and analysis of a sc e- 
nario by one or more SMEs. It is a modified ver- 
sion of a procedure described by McNeese and 
Zaff (1991) as design storyboarding. The tech- 
nique was used to help subject matter experts to 
“translate their conceptual knowledge and expe r- 
tise into a representation and design prototype 
that could be perceptually experienced by other 
viewers of the storyboard” (McNeese and Zaff, 
1991, p. 1184). 

The proposed method borrows from several 
existing KE methods. The method requires SMEs 
to identify specific knowledge requirements i m- 
posed on decision makers by analyzing a sc e- 
nario that illustrates a particular problem or pro c- 
ess. The advantage of this approach is that the 
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SMEs are asked to respond to the requirements 
of a specific context rather than a general, a b- 
stract problem. However, unlike the approach in 
which SMEs must interpret and respond to a pre- 
scripted scenario, the SMEs first construct the 
scenario and, thus, they are free to insert the 
events and conditions to which they will respond. 
The rationale behind this approach is that the 
SMEs will construct the scenario to reflect how 
they stereotypically conceptualize a problem, 
based on previous experiences and personal i n- 
sights. This procedure forces SMEs to exter- 
nalize at least part of their personal mental model 
of the problem. 

An additional advantage of this approach is 
that it builds on a task already performed by many 
TDM SMEs -- the construction of scenarios to 
facilitate specific training objectives. This proc e- 
dure attempts to formalize this familiar task, and 
forces the SMEs to identify the linkage between 
scenario events and knowledge r equirements. 

MAJOR KE TASKS 

The SMEs performs four major tasks in this 
procedure: 

1. The SME constructs a scenario illustrating 
a particular problem or procedure; 

2. The SME tests the logic of the scenario by 
identifying key variables in the scenario, and their 
relationships; 

3. The SME identifies the knowledge required 
of the decision maker to perform successfully in 
the scenario; and 

4. The SME provides graphical, auditory, and 
verbal representations of the key knowledge el e- 
ments. 

Constructing a Scenario: 

Specifying a Topic. The procedure focuses 
the SME on a single topic or problem to be illu s- 
trated. Examples of tactical problem or topics 
might include: managing tactical resources, 

dealing with equipment casualties, managing high 
workloads, or dealing with difficult environmental 
conditions. The SME can be asked to construct 
a scenario for a particular type of learner, such as 
a novice or an expert decision maker. Alte r- 
nately, the SME could be asked to develop a sc e- 
nario at a certain level of difficulty or complexity, 
or to emphasize a particular aspect of the pro b- 
lem -- teamwork versus individual perfor mance. 



Defining the scope of the topic is a critical 
step in the process. The narrower the scope, the 
more easily the SME can focus on the key var i- 
ables and relationships in the scenario. The 
broader the scope of the topic, the more complex 
the scenarios must be to encompass its full 
range. It may be easier for a SME to construct 
many specific, narrowly defined scenarios than to 
create a few very large, complex scenarios to 
illustrate all possible variations on a topic. 

Building a Background. Once the topic 
has been specified, the first step for the SME is to 
lay out a background for the scenario. The pu r- 
pose of this step is to provide a broad, common 
framework within which to fit specific events of 
the scenario. The framework identifies events and 
requirements that would normally occur in a see - 
nario such as shift changes, routine reports, or 
other operational cycles. 

The background should include the enviro n- 
ment in which the scenario will take place: ge o- 
graphic location; the geopolitical situation; the 
mission and status of one’s ownship; the order of 
battle; and any standing orders, taskings, rules, 
or procedures under which the ship is operating. 
Examples might include the Rules of Engag e- 
ment (ROE), that impose limitations on the sc e- 
nario. Unless these background elements will 
play a significant part in the scenario, they should 
be made as neutral or routine as possible. The 
background should not include variables that play 
a major part in the scenario or the way actors in 
the scenario should or will perform. These el e- 
ments should be explicitly considered in the key 
events of the scenario. 

Building Key Events. Once the scenario 
background has been specified, the next step is 
for the SME to insert events that will drive the 
illustration. It may be desirable to begin with a 
general outline of events in the scenario, and to 
then add detail in successive layers. At the end 
of this step, the scenario should include a detailed 
script of the scenario including descriptions of 
times, distances, locations, and behaviors of the 
actors, platforms and systems in the scenario. 
The steps might include... 

1 . Identifying key event or events that will illu s- 
trate the problem. 

2. Identifying key actors in each event, tasks 
they will perform, and actions they will take. 
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3. Identifying equipment, platforms, and sy s- 
tems that will be employed, or that will be a f- 
fected by the events. 

4. Describing the event in terms of locations, 
movements, and other behaviors (for example, 
communications, IFF, EW, etc.) 

Analyzing the Logic of the Scenario. 

After the scenario has been constructed, the 
next step is to verify the logic of the scenario to 
ensure that it represents the problem or proce- 
dure as the SME intends, and that the events 
contain a coherent set of identifiable variables 
and relationships. To verify the logic behind the 
scenario for each event the SME should be asked 
to 

• Identify important factors or variables in the 
event, and 

• Specify causal or logical links between the 
variables and the problem or process it is i n- 
tended to illustrate. 

For example, if an event is intended to illu s- 
trate the problem of determining the intent of an 
unknown track that is demonstrating erratic b e- 
havior, the SME would be asked to identify the 
variables that make the track’s behavior “erratic” - 
-e.g., sudden changes in altitude, frequent 
changes in course or speed. 

Second, the SME would be asked to identify 
the causal or logical relationship b etween the vari- 
ables-for example, erratic behavior indicates an 
unskilled pilot, mechanical difficulties, a possible 
diversion, etc. Obviously, it is important that the 
number of variables be kept to reasonable size, 
and that the SME focuses on the most important 
variables and relationships. The product of this 
step is a list of the key variables operating at each 
event in the scenario, and a description of the 
relationships among the key variables. 

Analyzing the Knowledge Requirements of the 
Scenario. 

The variables and relationships identified in 
the previous step form the basis for identifying the 
knowledge requirements for a decision maker 
performing in this scenario. The analysis could 
be performed in several ways. The SME could 
work through the scenario event by event, or 
variable by variable to identify specific knowledge 
requirements. To help the SMEs identify this 
knowledge, they might be prompted to focus on 
specific categories of knowledge. Working sy s- 



tematically through the scenario, the SME would 
follow a standard checklist of knowledge categ o- 
ries to ensure that they consider all relevant 
knowledge requirements. Within each knowledge 
category the SME would be asked to identify sp e- 
cific knowledge and skills needed by the decision 
maker to make a correct response to each event. 
An alternative approach might be to ask the SME 
to identify both correct and incorrect responses to 
events -- particularly common errors made by 
novice decision makers — and the outcomes of 
the events associated with each. 

Identifying Graphical and Verbal Representa- 
tions of Key Knowledge Elements. 

A major goal of this procedure is to provide 
the basis for creating media representations of 
the knowledge identified in the analysis. This 
goal could be pursued as a separate step after the 
knowledge requirements have been identified, or 
could be integrated into each of the previous 
steps. For example, during the scenario con - 
struction step, the artifacts of the process could 
be used as the basis for representing the know I- 
edge elements identified earlier. This could be 
done by asking the SME to identify the external 
indicators and cues associated with each major 
event, or each variable in the scenario. Sim i- 
larly, during the second step, when the SME 
identifies the key variables and relationships he 
could be asked to identify the external indicators 
that allow the decision maker or operator to know 
that a variable’s value or state has changed. In 
the last step in the procedure the SME could be 
asked to specify the observable external indie a- 
tors of correct and incorrect performance (e.g., 
voice phraseology, keypress sequence, balltab 
movement, etc.) It may be possible to solicit 
sketches, diagrams, maps, or other illustrations 
from the SME that can be directly translated into 
media objects by a graphic artist or illustrator. 
Similarly, verbal protocols that represent voice 
communications between persons during the see - 
nario should be collected from the SME. 

This procedure does not assume that one 
scenario can capture all possible aspects of a 
problem or procedure, nor does it assume that 
one SME will necessarily understand all aspects 
of the problem equally well. The procedure co n- 
templates the need for multiple scenarios and 
multiple SME input. It is also possible for the 
process to be carried out by a team of SMEs 
working together as a group. However, the re - 
quirements on the SMEs are thought to be no 
greater than that imposed by other KE proce- 
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dures, and given the nature of the task, may be 
easier for the SME to understand what is re- 
quired. 

SEA STORIES: AUTOMATING THE PROCESS 

Parts of this KE procedure could be signif i- 
cantly improved by providing automated tools to 
accelerate the construction of the scenario, co m- 
pleting the analysis of the scenario’s logic, and 
recording the knowledge requirements of the sc e- 
nario. Such a device would have the capability of 
depicting all elements of a typical tactical see - 
nario in a naturalistic format, and ordering and 
correlating the SME’s inputs to produce a cohe r- 
ent scenario timeline and consistent, integrated 
knowledge structure. 

Sea Stories Prototype 

Describing the different types of knowledge 
involved in a scenario requires several different 
abstractions which may be depicted using a range 
of media types. Creating and editing these de- 
pictions requires a suite of tools, most of which 
are available as commercial-off-the-shelf (COTS) 
software products. The design and development 



of Sea Stories will leverage these commercial 
tools and provide domain experts with a high- 
level integrating framework for composing know I- 
edge elements. The contribution of Sea Stories is 
to integrate and organize these depictions into a 
complete description of tactical knowledge. 

Starting from the mental models associated 
with TDM identified by Search Technology 
(Duncan, Rouse, Cannon-Bowers, Salas, John- 
ston, & Burns, 1996), such a tool would enable 
the SME to depict events in the following categ o- 
ries: 

• Situation, 

• Tasks, 

• Team, 

• Equipment & Systems, and 

• Ship/Group. 

The Situation interfaces would permit the 
SME to depict scenario events within both the 
external environment and the internal enviro n- 
ment of the Combat Information Center (CIC). 
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Figure 2. The External “Situation" Interface 



This depiction would include, for example, 
the physical location of platforms, their heading, 
speed, altitude; their detectable behaviors such 
as electronic emissions, communications, 
changes in course or altitude, weapon release, or 
any other changes that materially affect the sc e- 
nario. The Situation interfaces would allow the 
SME to place the platforms and specify their 
performance characteristics and behaviors at 
each point in the scenario. The interfaces should 
also permit the SME to depict the influence of 
environmental factors in the scenario such as 
weather, time of day, geophysical e ffects. 

Figure 1 shows a high level depiction of the e x- 
ternal "situation” knowledge in a story. In this sto- 
ryboard, the depiction of the situation is simplified 
to include only the most important entities in the 
story — two threatening aircraft and ownship. The 
clock in the upper right corner of the screen i n- 
dexes the frame of the story as a time offset from 
the beginning of the scenario. The number of 
these "key-frames" is dependent upon the co m- 
plexity of the story and the number of key events 
in the scenario. 



The dialogue window in the middle of Figure 
1 illustrates a key aspect of the computer-based 
tool. Sea Stories is essentially a detailed data- 
base of knowledge elements with a variety of vi s- 
ual representations serving as stimuli to aid the 
analysts in articulating the details of the know I- 
edge. This dialogue window shows the current 
record in the database pertaining to one of the 
aircraft in the storyboard during this event in the 
scenario. Details about the aircraft can be inter - 
actively defined and the dynamic values can be 
updated for different points in time in the story. 
The details include both “ground truth" knowledge 
(i.e., knowledge that ownship cannot be certain 
of, such as the threatening aircraft's role) and I o- 
cal knowledge (i.e., ownship's data readouts 
about the track). One important aspect of the 
power of Sea Stories comes from the capability 
provided to the analyst to query the database to 
access and compare knowledge elements at di f- 
ferent levels of abstraction and at different points 
of time in the story. 

Figure 2 shows another visual depiction of 
the situational knowledge from the perspective of 
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Figure 3. The “Systems” Interface 



ownship’s CIC. Ownship’s picture of the situation 
is more complicated and more abstract than the 
simplified "ground truth" version presented in Fig - 
ure 1. The presence of many other tracks reflects 
the real complexity of the environment, and it b e- 
gins to introduce the difficult nature of the visual 
task of picking the signal -- the significant events 
in the story -- from the noise. 

The dialogue window in the middle of Figure 
2 shows the detailed information about the track 
of interest at this point in the story from the per - 
spective of the CIC. The track of interest is re p- 
resented in this depiction by the highlighted sy m- 
bol just to the left of the popup window. This i I- 
lustrates how Sea Stories connects the underl y- 
ing data to related storyboards, and how different 
storyboards can be accessed from a single data 
point. 

The Systems interfaces would allow the 
SME to depict the key equipment- and systems- 
related events in the scenario. For example, the 
Systems interfaces should depict factors relating 



to the operational performance of ownship dis- 
plays, the capabilities and limitations of the ship’s 
sensors or weapons, the lag between a track’s 
actual change in course and speed and its dete c- 
tion by the SPY radar. The interface may include 
the ability to depict the physical effects, both vis i- 
ble and invisible, that underlie the operational 
characteristics of a system or piece of equipment, 
such as the effect of distance on a radar signal, or 
how changes in the environment affect different 
sensors. 

Figure 3 shows an example storyboard diagram 
of System knowledge for a particular point in the 
story. The Figure shows the high-level system 
components and the interconnections among 
them. The graphical annotations are linked to 
database records that describe or explain how the 
system interactions are important at this point in 
the story. For example, in response to a new type 
of threat or a new operating environment, a sy s- 
tem may need to be set up in a non-standard 
mode. 
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The Task Interfaces permit the SME to depict 
the specific procedures that are associated with a 
particular scenario. This would include the activ i- 
ties of different watchstations in the CIC or other 
major players in the scenario. The procedures 
might be related to console operations, commun i- 
cations, application of the ROE, weapon alloc a- 
tion and use, or setting shipboard conditions. The 
Task interfaces also will include planning tasks 
such as positioning assets, pre-planned r e- 
sponses, or setting tripwires. The level of detail 
at which a task can be described will vary. For 
example, the SME might say, “at time X the AIC 
should vector the CAP to intercept the incoming 
track at distance Y from ownship," without de- 
scribing what this task actually involves. Alte r- 
nately, the SME might describe this task in detail 
because the scenario requires a variation in the 
way the task is normally done. The interface 
would allow the SME to decompose the tasks to 
the level necessary to capture the significant 
factors in the scenario. 

The Team interfaces would be similar to the 
Task interfaces, except that the nature of the 



tasks should be at the team rather than the ind i- 
vidual watchstation level. Team-level tasks focus 
on tasks such as communications, team leader - 
ship and initiative, and workload allocation. Fi g- 
ure 4 illustrates the Team knowledge representa- 
tion screen. The example shows, through an a n- 
notated visual picture, a pattern of communic a- 
tion that should occur at a particular moment 
when a team is responding to the threat described 
in the story. If the nature of the communication 
justified it, audio depictions could also be used to 
illustrate this type of know ledge. 

In a similar manner, other knowledge types 
can be described. Knowledge about a particular 
item of equipment could be illustrated from a line 
drawing or a photograph, and annotated using 
simple graphic tools linking visual annotations 
(e.g., lines, arrows, or boxes) to elements in the 
database. A task description might be illustrated 
and annotated using a flow chart. 

Collaborative Work 

As we suggested earlier, this procedure does not 
assume that knowledge regarding a particular 
tactical problem resides with one individual. Sea 




Sea Stories 



File EcB lent gage Help 




Situation: Low Level Popup with Decoy 



'i 'v__- 



\\ y 






ii ^ 



o a o 



Question: Is this track extraneous? 



Source: LCD Kelly 
ATGPAC 

06 Mar 97 08:30 
Replies: 5 total, 3 unread 



Q: 06 Mar 97 08:30 LCD Kelly 
Y: 07 Mar 97 08:54 CMD Smith 
Y: 07 Mar 97 10:47 LCD Minsk 
N: 1 0 Mar 97 07:32 LCD Lassiter 

Re: track 7025: It seems that this 
track only confuses the scenario. It 
only makes a difference in a really 
convoluted way. I think we should 
keep this focused on the higher 
priority threats - I suggest we 
delete this one. 



Figure 5. The SME Collaboration Interface 




Disposition: replies due by 14 March 97 
Your disposition: (pending) 

] 



Stories facilitates the collaboration of a team of 
SMEs to cover the landscape. Frequently SMEs 
are in different locations and are available to work 
on the analysis at different times. The pervasive 
presence of computers and the rapid growth of 
wide area networks makes it easier and faster for 
persons to collaborate despite the barriers of time 
and distance. The field of computer science 
known as Computer Supported Collaborative 
Work (CSCW) has grown to study the promises 
and problems of these technologies. Commercial 
applications abound, from single-function tools to 
integrated desktop tools. 

Figure 5 shows a concept for CSCW cap a- 
bilities within Sea Stories. In this example, a 
team member has raised a question about a par - 
ticular track, and the other team members carr y- 
ing on a discussion about the point using ele c- 
tronic mail. In the illustration, the electronic mail 
messages are linked with objects in the know I- 
edge base, providing a flexible means for ac - 
cessing and reviewing all the group's discussion 
issues related to, for example, a track or a wor k- 
station. 



Along with context sensitive dialog, other 
important CSCW capabilities include maintaining 
an "institutional memory" by maintaining altern a- 
tive storyboards, tracking who made revisions 
when, and providing personalized summaries of 
changes on request. 

Using CSCW techniques within Sea Stories 
has some potential pitfalls. Research has shown 
that "depictional lock-on" sometimes occurs 
when developing design artifacts using CSCW. 
Depictional lock-on is the tendency of users to 
eventually focus on the depiction of the artifact 
rather than the goal of the analysis ( Whitaker, 
Selvaraj, Brown, & McNeese, 1995). Given the 
strongly visual nature of Sea Stories artifacts, 
this could be a potential weakness, although it is 
certainly not unique to computer-based tools or 
CSCW. On-going CSCW research should be 
followed for guidelines and lessons-learned that 
can be applied to manage this problem for Sea 
Stories. 

OPEN ISSUES FOR RESEARCH 

Besides the CSCW issues, there are two 
other research issues that will influence the su c- 




cess of Sea Stones as a computer-based know I- 
edge articulation and analysis tool. The first issue 
relates to the flexibility (or the lack of flexibility) 
that pre-defined knowledge frameworks introduce. 
Will pre-defined frameworks constrain or bias 
knowledge elements? Can the knowledge 
frameworks within Sea Stories be flexible enough 
to accommodate radically different types of 
knowledge, or will they "force" the analysis along 
certain pre-defined paths? 

As initially conceived, Sea Stories will pro- 
vide a great deal of flexibility in defining and r e- 
fining the knowledge frameworks. However, there 
will be a trade-off between flexibility and the 
guidance and/or on-line help that will be available 
to the users. For Sea Stories to provide signifi- 
cant help for users (experts in domain knowledge 
but not necessarily in this type of analysis), it 
must have a guiding framework. Pre-defining and 
providing that framework may constrain the user 
from doing certain things. A proper balance be - 
tween these questions will need to be studied. 

The second issue is the cost/benefit ratio that 
the user perceives in using the tool. Analysis is 
by its nature a difficult and tedious task that can 
be accomplished with ordinary tools (e.g., paper, 
pencil, telephone, voice mail, electronic mail, and 
fax machines). If a tool adds a layer of comple x- 
ity to that task, the benefit of dealing with that 
complexity must be readily perceived. 

The promised payoff in the Sea Stories con- 
cept is 1) the ease of entering, organizing, and 
accessing the knowledge prompted by creating 
the storyboards and 2) the opportunity to bring 
several highly qualified people in on the analysis 
without having to travel or coordinate schedules. 
Insofar as these promises can be realized, a d e- 
gree of inconvenience will be acceptable. 
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